Papers with multimodal retrieval
Normalized Contrastive Learning for Text-Video Retrieval (2022.emnlp-main)
Copied to clipboard
| Challenge: | Cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. |
| Approach: | They propose a normalized contrastive learning algorithm that normalizes the sum retrieval probabilities of each instance so that every text and video instance is fairly represented. |
| Outcome: | Empirical results show that NCL brings significant gains in text-video retrieval on different model architectures without any architecture engineering. |
UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | e-commerce tasks such as multimodal retrieval and multimodal generation are largely ignored due to the diversity of the multimodal fashion domain. |
| Approach: | They propose a framework that integrates image generation with retrieval and text generation tasks. |
| Outcome: | The proposed framework outperforms state-of-the-art models across fashion tasks. |
RA-RRG: Multimodal Retrieval-Augmented Radiology Report Generation with Key Phrase Extraction (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing MLLMs are computationally expensive and may produce hallucinated content . RA-RRG uses large language models to generate radiology reports . |
| Approach: | They propose a retrieval-augmented RRG framework that combines multimodal retrieval with large language models to generate radiology reports. |
| Outcome: | RA-RRG uses large language models to generate radiology reports . it suppresses hallucinations while maintaining strong report generation performance . |
Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval? (2026.findings-acl)
Copied to clipboard
| Challenge: | Rapid advances in multimodal large language models have revolutionized cross-modality understanding. |
| Approach: | They propose a method that uses whitening transformations to adjust MLLM representation spaces . they propose ML models that are dominated by textual semantics and visual semantics . |
| Outcome: | The proposed approach improves zero-shot multimodal retrieval performance without fine-tuning efforts. |
Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Visual word sense disambiguation (VWSD) is a challenging task involving multiple candidates . context given for an ambiguous word is minimal, most often limited to a single word . |
| Approach: | They propose to use large language models to enhance given phrases and resolve ambiguity related to the target word. |
| Outcome: | The proposed frameworks improve the image representation of ambiguous words among candidates and achieve competitive ranking results. |
MegaPairs: Massive Data Synthesis for Universal Multimodal Retrieval (2025.acl-long)
Copied to clipboard
Junjie Zhou, Yongping Xiong, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian
| Challenge: | despite the growing demand for multimodal retrieval, there is a lack of training data. |
| Approach: | They propose a data synthesis method that leverages vision language models and open-domain images to generate high-quality data. |
| Outcome: | The proposed method outperforms baseline models on 70 more datasets and can scale up. |
Enhancing Multimodal Retrieval via Complementary Information Extraction and Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on capturing information in multimodal data that is similar to their paired texts, but often ignores the complementary information contained in multimodule data. |
| Approach: | They propose a multimodal retrieval approach that employs Complementary Information Extraction and Alignment to capture complementary information in multimodal data. |
| Outcome: | The proposed approach achieves significant improvements over divide-and-conquer models and universal dense retrieval models. |
OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for Knowledge-Based Visual Question Answering lack multimodal retrieval . large language models (LLMs) have demonstrated remarkable generalization and reasoning capabilities in text-based systems. |
| Approach: | They propose a multimodal vision-language retrieval-augmented generation system that harmonizes multiple modalities and modality to enhance retrieval. |
| Outcome: | The proposed system achieves state-of-the-art retrieval performance and competitive answers on InfoSeek and Encyclopedic-VQA benchmarks. |
Text-to-Multimodal Retrieval with Bimodal Input Fusion in Shared Cross-Modal Transformer (2024.lrec-main)
Copied to clipboard
| Challenge: | Multimodal video retrieval systems are needed for multimodal content retrieval . multimodal video search systems are sub-optimal for multi-modal content representations . |
| Approach: | They propose a model that learns retrieval cues for the textual query from multiple modalities and a shared embedding space with task-specific contrastive loss functions. |
| Outcome: | The proposed model outperforms state-of-the-art methods on the MSR-VTT and YouCook2 datasets and shows significant improvements from baseline. |